Keywords, Hashtags or Account Lists? Choosing How to Sample Public Social Media Data

Before collecting any public social media data, a project has to decide how posts get into the dataset. The four common choices are keyword search, hashtags, a curated list of accounts, and a sample provided by the platform itself through an official API or research program. Keywords give the broadest view of a conversation but bring the most noise. Hashtags are precise but only capture people who tag. Account lists are the most reproducible but miss everyone outside the list. Platform samples come closest to representative where they exist, but you control the least. The right choice depends on the question the data has to answer, and many projects end up combining two methods.

The criteria that decide the choice

Each method below is compared against the same seven criteria, so the trade-offs line up.

• Coverage: how much of the relevant conversation the method can reach.
• Bias: which kinds of users and posts the method over- or under-represents.
• Noise: how many collected posts turn out to be irrelevant.
• Volume control: how predictable the number of posts is, which drives cost and storage.
• Reproducibility: whether someone else could rebuild a comparable dataset from your description.
• Fit with access rules: whether the method works within the platform's official API or published terms.
• Personal data exposure: how much information about private individuals the method pulls in.

Write these criteria down before looking at options. Teams that skip this step tend to pick the method that is easiest to start, then discover months later that it cannot answer the question they were funded to answer.

Option 1: Keyword search

Keyword collection selects posts that contain one or more terms, usually through a platform's search function or an official API search endpoint. It reaches people who talk about a topic without tagging it, which makes it the method with the widest coverage of everyday conversation.

The cost is noise. Common words carry several meanings, brand names collide with ordinary words, and sarcasm or quotes pull in posts that mention a term while discussing something else. Coverage is also less complete than it looks: search results on many platforms are ranked, filtered or limited in time, so "every post containing the word" is often not what comes back. Language matters too, since a keyword list in one language silently excludes the rest of the conversation.

Concrete check: run the keyword list for a short pilot period and read a random sample of the results. Suppose, as a hypothetical, 200 posts are read and 120 are on topic: the list has a precision of about 60 percent. Adding exclusion terms and re-checking a fresh sample shows whether the precision improves without losing obvious relevant posts.

Option 2: Hashtags

Hashtag collection selects posts that carry one or more specific tags. For events, campaigns and organized movements, it is precise: someone who adds an event's tag is almost always talking about that event.

The bias is strong, though. Hashtag users are a subset of the people discussing a topic, and often the more engaged, organized or promotional subset. Tags are also hijacked: a popular tag attracts spam and unrelated posts trying to ride its visibility. And tags change during an event, as people shorten them, misspell them or switch to a competing one.

Concrete check: during the pilot, list the other hashtags that appear most often alongside your seed tags. Tags that show up repeatedly and clearly refer to the same topic are candidates to add; tags that point to spam or unrelated content tell you what to filter. Record every change to the tag list with a date, because a dataset whose tag list changed mid-collection is not comparable before and after the change.

Option 3: Curated account lists

Account-based collection follows a defined list of public accounts and gathers what they post. It suits questions about specific actors: how a set of companies talk to customers, how public agencies communicate during an emergency, or how news outlets cover a story over time.

It is the most reproducible method, because the list itself can be published and reused, and its volume is the most predictable. Its limit is equally clear: it says nothing about anyone outside the list, and the list carries the bias of whoever built it. Lists also decay as accounts are renamed, deleted or made private.

The type of account matters for privacy. A list of organizational and official accounts raises far fewer questions than a list of private individuals. Following named individuals, even with public profiles, starts to look like monitoring people, which carries much more weight under privacy law and research ethics, and generally needs a clear justification and legal review.

Concrete check: document how the list was built, with a written inclusion rule such as "the official accounts of all public transit agencies in the region", and re-check the list at a fixed interval to catch renamed or closed accounts.

Option 4: Samples provided by the platform

Some platforms have offered sampled data through official APIs or research access programs, for example a sample of public posts over a period or access designed for approved researchers. Where such access exists, it is the option closest to a representative picture of activity on the platform, because the selection does not depend on your choice of words, tags or accounts.

The trade-offs are about control and continuity. Availability, eligibility, volume limits and sampling methods are set by the platform and have changed over time, sometimes at short notice. The sampling procedure may not be fully documented, which limits what you can say about representativeness. And access usually comes with its own terms on storage, sharing and deletion.

Concrete check: before relying on a platform sample, read its current documentation for how the sample is drawn, what is excluded, and what happens to your access and stored data if the program ends.

Side-by-side summary

Fit with access rules is left out of the table because it depends on each platform: check whether the method you choose is supported by the official API or allowed by the published terms before comparing anything else.

Running a pilot that compares two methods

When two methods both look plausible, a short pilot that runs them side by side settles the question better than any argument. Keep the pilot small, within the access route you plan to use, and the same length for both methods, ideally covering a full week so weekday and weekend patterns both appear.

Suppose, as a hypothetical, a team studying public reactions to a regional transit fare change runs a keyword list and a hashtag list in parallel for seven days. The keyword stream returns about 4,000 posts and the hashtag stream about 900. Reading a random sample of 200 from each, the team finds 110 relevant keyword posts and 180 relevant hashtag posts. The keyword stream is noisier, at roughly 55 percent precision against 90 percent, but in absolute terms it still contains far more relevant posts: around 2,200 against around 810.

The overlap tells the rest of the story. If only a small share of the relevant keyword posts also carry the tag, the hashtag stream is capturing a distinct, more organized slice of the conversation, and relying on it alone would miss most ordinary riders. The team in this example would likely keep keywords as the main method, invest in exclusion terms to raise precision, and keep the hashtag stream as a labeled secondary source.

Two cautions apply to any pilot. A week can be unusual, so note any event that might distort it. And the relevance judgments should follow a written definition, ideally checked by a second reader on part of the sample, because "relevant" drifts when one person codes hundreds of posts.

Recording the sample so others can rebuild it

Whatever method you choose, the dataset is only interpretable if the way it was built is written down. A short sampling record should include:

• The exact selection rules: the full keyword and exclusion lists, the hashtags, or the account list with its inclusion rule, each with the dates it was in force.
• The access route: which official API, research program or public pages were used, and under which terms.
• The collection window and time zone, so "posts from March" means the same thing to every reader.
• The collection lag: how long after publication posts were collected. Posts deleted or edited before collection never appear, and a longer lag means more of them are missing.
• Known gaps: outages, rate-limit pauses and days with partial collection.

This record costs an hour to write and saves weeks of confusion when someone asks, months later, why two reports built on "the same data" disagree.

Combining methods without muddling the dataset

Many projects use two methods: an account list for the actors that matter most, plus keywords to see how the wider conversation reacts. This works well as long as each post records which method brought it in. Without that field, an analyst cannot tell whether a spike came from the accounts or from the keyword stream, and the two samples end up mixed in ways no one can undo.

A second rule: deduplicate across methods but keep all the labels. A post found both by keyword and by account list is one post with two origins, not two posts. The guide to deduplicating scraped records covers keys, normalization and near-duplicate checks.

Mistakes that undermine the sample

• Treating search results as complete. A keyword search returns what the platform's search returns, which may be ranked, filtered or capped. Describe the dataset as "posts returned by search for these terms", not "all posts about the topic".
• Changing the query without a log. Every term, tag or account added or removed should be recorded with a date and a reason, so later analysis can account for it.
• Skipping the pilot. A week of pilot collection and a read-through of a random sample reveals most noise and coverage problems before they are baked into months of data.
• Collecting more profile data than the question needs. Sampling method and data minimization go together: if the question is about topics, the dataset rarely needs follower lists or profile details. The overview of ethical and effective social media scraping best practices covers this in more depth.

Which method fits which project

Brand and product teams tracking reactions: start with keywords, because customers rarely use official tags. Budget time for the pilot and for exclusion terms, and report precision alongside the findings.

Researchers who need representative claims about a platform: use a platform-provided sample or research access where it exists and fits your project, and be explicit about its documented limits. Where it does not exist, narrow the claim to what keywords or account lists can support.

Communications and policy teams following specific organizations: use an account list of organizational accounts, published with its inclusion rule. It is predictable, low in personal data and easy to repeat next year.

Teams covering a live event or campaign: start with the event's hashtags for precision, add keywords for the people who do not tag, and label each post by method.

Whichever method you choose, the sampling decision belongs to the project, not to the tool that runs the collection. If a managed provider such as ScrapingIsNotACrime handles the collection, hand over the written criteria, the term or account list and the change log as part of the specification, so the dataset that comes back matches the question.