Welcome to Mastodon Scraper ## 🐘 About Mastodon (social network) Mastodon is free and open-source software for running self-hosted social networking services. It has microblogging features similar to Twitter, which are offered by a large number of independently run nodes, known as instances, each with its own code of conduct, terms of service, privacy policy, privacy options, and content moderation policies.[6][7][8] ## 🐘 About This Actor Scrape data from Mastadon, the decentralized alternative to Twitter. Extract information on users, their "toots," likes, and favorites or scrape data based on a search query, such as a hashtag or a keyword. Export accumulated data into a variety of database formats. ## 🐘 Tutorial ### 📁 Parameters | Input| Type | Description | |-|-|-| | query | string array | List of Query, Search Terms or Start URL | | portal | string | Default portal (default: mastodon.social) | | limit | integer | Number of results (per-query) | ### 📁 Basic Usage yaml # search posts { "query": ["sponge bobs"], "limit": 10 } yaml # search hashtag { "query": ["#python"], "limit": 10 } yaml # just paste an url { "query": ["https://mastodon.social/@elonmusk"], "limit": 10 } ### 📁 Advanced Usage Mastodon Query Language (MQL) html [ [ <COMMAND:> | # | @ ] [ <NAME> | <ID> | <KEYWORD> ] [ / <SECTION> ] | <URL> ] Possible QUERY values | Format | Example | Description | |-|-|-| | KEYWORDS | search terms |Search Anything| | KEYWORDS |webscraping| Search posts | | users:KEYWORDS |users:python| Search users | | hashtags:KEYWORDS |hashtags:php| Search hashtags | | portal:SECTION | | Explore Current Portal| | portal:about | | Portal info & description | | portal:info | | Portal info | | portal:description | | Portal description | | portal:posts | | Portal posts | | portal:hashtags | | Portal hashtags | | portal:news | | Portal news | | portal:feeds | | All feeds | | portal:local | | Portal feeds (local) | | portal:remote | | Portal feeds (remote) | | #HASHTAG | #python | Scrape Hashtag | /info | #python/info | Hashtag info | | /posts | #python/posts | Hashtag posts | | @USER[@SERVER] | @elonmusk | Scraper User | | | @elonmusk@mastodon.social | | | /info | @Gargron/info | User info | | /posts | @Gargron/posts | User posts | | /replies | @Gargron/replies | User posts (with replies) | | /media | @Gargron/media | User media | | /following | @Gargron/following | User following | | /followers | @Gargron/followers | User followers | | | 112718231305707672 | Scraper a Post | |/info | 112718231305707672/info |Post content | |/comments | 112718231305707672/comments | Post comments | |/reblogs | 112718231305707672/reblogs | Post reblogs (boosts) | |/likes | 112718231305707672/likes | Post liked users (favourites) | | https: | https://mastodon.social | Start URL | | https://mastodon.social/@elonmusk | ### 📁 Examples By default the actor will scrape from mastodon.social portal (eg: https://mastodon.social) Example 1: Portal To scrape from other portal, use portal parameter and specify the website/portal hostname : yaml { "portal": "fosstodon.org" } Example 2: Scraping #hastag yaml { "query": "#microsoft" } Example 3: Scraping user posts yaml { "query": "@username" } { "query": "@username@portal" } Example 4: Using URL as input When you specify URL for query parameter, it will automatically parsed with appropriate parameter : yaml { "query": "https://fosstodon.org/@Joefree/media" } Example 5: Number of results Use limit parameter to get more results. yaml { "query": "microsoft", "limit": 100 } ## 🐘 Output Examples yaml { "account": { "acct": "Gargron", "avatar": "https://files.mastodon.social/accounts/avatars/000/000/001/original/dc4286ceb8fab734.jpg", "avatar_static": "https://files.mastodon.social/accounts/avatars/000/000/001/original/dc4286ceb8fab734.jpg", "created_at": "2016-03-16T00:00:00.000Z", "discoverable": true, "display_name": "Eugen Rochko", "fields": [ { "name": "Patreon", "value": "<a href=\"https://www.patreon.com/mastodon\" target=\"_blank\" rel=\"nofollow noopener noreferrer me\"><span class=\"invisible\">https://www.</span><span class=\"\">patreon.com/mastodon</span><span class=\"invisible\"></span></a>" }, { "name": "GitHub", "value": "<a href=\"https://github.com/Gargron\" target=\"_blank\" rel=\"nofollow noopener noreferrer me\"><span class=\"invisible\">https://</span><span class=\"\">github.com/Gargron</span><span class=\"invisible\"></span></a>", "verified_at": "2023-02-07T23:24:40.347+00:00" } ], "followers_count": 312471, "following_count": 395, "header": "https://files.mastodon.social/accounts/headers/000/000/001/original/3b91c9965d00888b.jpeg", "header_static": "https://files.mastodon.social/accounts/headers/000/000/001/original/3b91c9965d00888b.jpeg", "id": "1", "last_status_at": "2023-04-28", "note": "<p>Founder, CEO and lead developer <span class=\"h-card\"><a href=\"https://mastodon.social/@Mastodon\" class=\"u-url mention\">@<span>Mastodon</span></a></span>, Germany.</p>", "statuses_count": 73541, "url": "https://mastodon.social/@Gargron", "username": "Gargron" }, "application": { "name": "Web" }, "card": { "blurhash": "U34x7la*9EoYxNfQayfR4mj:?cWIajfRj[a{", "description": "Learn how to get verified on Mastodon", "height": 267, "image": "https://files.mastodon.social/cache/preview_cards/images/059/998/839/original/74e91efacf98def0.png", "title": "Verification", "type": "link", "url": "https://joinmastodon.org/verification", "width": 400 }, "content": "<p>Looks like <span class=\"h-card\"><a href=\"https://mastodon.social/@arstechnica\" class=\"u-url mention\">@<span>arstechnica</span></a></span> is now verified on here! Always great to see this feature used. It'll be even more prominent in the future.</p><p><a href=\"https://joinmastodon.org/verification\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><span class=\"invisible\">https://</span><span class=\"\">joinmastodon.org/verification</span><span class=\"invisible\"></span></a></p>", "created_at": "2023-04-27T22:37:39.080Z", "favourites_count": 420, "id": "110273171231251728", "language": "en", "mentions": [ { "acct": "arstechnica", "id": "110266162634306901", "url": "https://mastodon.social/@arstechnica", "username": "arstechnica" } ], "reblogs_count": 280, "replies_count": 14, "uri": "https://mastodon.social/users/Gargron/statuses/110273171231251728", "url": "https://mastodon.social/@Gargron/110273171231251728", "visibility": "public" } ## 🐘 Support