Skip to content
Home Posts About us Privacy Policy
Posts Only, Posts with Replies or Full Threads: How Much of a Conversation to Collect

Posts Only, Posts with Replies or Full Threads: How Much of a Conversation to Collect

Share ๐• in f W

The decision is how deep into a public conversation your collection should go: engagement counts only, the original posts only, posts with their direct replies, or complete threads with every branch. Each step deeper adds context and adds cost in three currencies at once: requests to the platform, people brought into the dataset, and work to keep the data in sync later. The right depth depends on the question you are trying to answer, not on what is technically available, and the most common mistake is collecting full threads because they are there and then analyzing only the original posts. Below, the four options are compared against the same six criteria, followed by a recommendation for each type of project.

The criteria used for every option

Volume and request cost: how many items and how many requests per original post. Context: whether the data lets you tell what a post meant and how it was received. People in the data: how many accounts end up in the dataset and how far they are from the public-facing role of the original author. Access route: whether official APIs, researcher programs or public pages make the option practical under platform terms. Sync burden: how much changes after collection, through deletions, edits and late replies, and what it costs to keep up. Analysis fit: which questions each depth can and cannot answer. The fourth criterion is where projects most often go wrong, so it gets the most attention below.

Option 1: engagement counts only

Here you collect each original post plus its counters: replies, reposts, likes, views where the platform shows them. No reply text, no replier identities. Volume is one item per post and usually one request per page of posts. Context is thin: you know a post was discussed, not what was said. People in the data are limited to the authors you chose to follow, who are typically public accounts, brands or officials posting for an audience. Access is the easiest of the four: most official APIs return counters with the post, and public pages show them.

The sync burden is real but cheap: counters keep changing for days, so a single collection gives a snapshot at an arbitrary moment. A project that compares posts needs a fixed window, such as counters at forty-eight hours after posting, which means one revisit per post at a known time. Analysis fit covers reach and relative attention: which topics draw replies, how attention decays, which accounts get amplified. It cannot say whether the replies agreed, disagreed or were spam.

The concrete check for this option is the timing rule. Suppose a hypothetical set of two hundred public accounts posting about forty times a month each: that is eight thousand posts a month and, with one revisit per post at the forty-eight-hour mark, sixteen thousand requests in total, spread evenly and well within what a polite schedule allows. Without the revisit, the dataset mixes counters captured ten minutes after posting with counters captured a week later, and any comparison between posts measures collection timing rather than attention.

Option 2: original posts only

This option collects the full text, media references, timestamps and author of each original post, with counters as in option 1 but without any reply content. Volume stays at one item per post. Context improves: you have what was said. People in the data remain the chosen authors. Access is as easy as option 1 and is the depth most platform terms and API tiers are designed around.

Sync is manageable: posts get deleted or edited, and the dataset needs a rule for what happens then; the post on deleted and edited posts in social media datasets covers the choices. Analysis fit covers everything about what accounts say: topics, timing, language, posting frequency, links shared. It fails for reception questions. Suppose a hypothetical study of how fifty city transit agencies announce service disruptions. Posts only answers how quickly and in what format they announce; it cannot answer whether riders found the announcements clear, because the riders' reactions are in the replies.

What to store at this depth deserves a decision of its own. A post often carries a quoted or reposted item, a link preview and media attachments; storing the identifiers of those items, rather than fetching each one, keeps the request count at one per page of posts while preserving the ability to follow the reference later if a question requires it. The sampling choice, whether by keyword, hashtag or account list, shapes this option more than any other, and the trade-offs are compared in choosing how to sample public social media data.

Option 3: posts with direct replies

Now each original post is collected together with the replies made directly to it, but not replies to those replies. Volume jumps and becomes unpredictable: a quiet account may average two replies per post and a controversial one two hundred, and each post typically needs at least one extra request, more for long reply lists with pagination. Context becomes rich enough for reception questions: tone of replies, common complaints, whether the author answered.

People in the data change character here, and this is the criterion that should decide most projects. Repliers are usually ordinary accounts, not public figures, replying in the moment to a post they saw. Their replies are public, but their expectation of being compiled into a dataset is weaker than that of the brand or official who posted. Under privacy law, this shifts the balance in any legitimate interests reasoning and widens the set of people who could make a rights request. The seven questions in what to answer before collecting public social media data for research apply with more force at this depth.

Access becomes uneven: some official APIs return replies in a separate endpoint with its own quota, researcher programs often allow it with conditions, and public pages may show only a subset of replies without interaction. Sync burden multiplies: replies arrive for days after the post, individual replies get deleted far more often than original posts, and a reply whose parent was deleted becomes an orphan. Analysis fit now covers reception, response time by the author, and the share of replies that are questions, complaints or spam, as long as the analysis respects that the reply sample was captured at one moment.

Two practical rules keep this option under control. First, cap the number of replies collected per post, for example the first two hundred in platform order, and record both the number seen and the number collected so that the cap is visible in the data rather than hidden. Second, store the reply's relationship to the post explicitly, with the parent identifier and the capture time, so that a reply collected during a later revisit can be told apart from one collected with the post. In the transit example, these two rules turn an open-ended collection into a fixed budget: fifty agencies, roughly three posts a day each, at most two hundred replies per post, revisited once, which is a number you can put in a plan.

Option 4: full threads with every branch

The deepest option follows every reply chain to its end: replies to replies, quoted reposts with their own replies, and the branching that results. Volume is the largest and the least predictable: a single viral post can produce a thread larger than a month of an ordinary account's activity, and collecting it requires repeated paginated requests across many branches. Context is complete: you can reconstruct who answered whom, where arguments broke out, and how a correction propagated.

People in the data are now the widest set, including accounts several steps removed from the original author who may have been replying to a friend rather than to the public post. Access is the hardest: full thread reconstruction often exceeds API quotas, and public pages rarely expose deep branches without interaction. Sync is close to impossible to keep perfect; threads change shape continuously and a reconstruction is only ever valid for the moment it was taken. Analysis fit is unique, though: conversation structure, cascade shapes, argument dynamics and moderation effects can only be studied at this depth.

In the hypothetical transit agency study, full threads would let you see whether riders answered each other's questions before the agency did. That is a legitimate question, but it is a different study from the one about announcement practice, with its own ethics review, a different sampling plan and roughly an order of magnitude more data.

If full threads are needed, sample threads rather than accounts. Choosing a fixed number of threads per account per week, or threads above a minimum reply count, keeps volume bounded and makes the collection reproducible, which an open crawl of every branch never is. Record the depth reached in each branch and whether the branch ended because the conversation ended or because a limit was hit; an analysis of conversation length is meaningless if truncated branches are counted as complete.

How the options compare on the six criteria

On volume, counts only and posts only are flat and predictable; posts with replies grows with popularity; full threads grows without a reliable ceiling. On context, each step adds a layer, with the largest jump between posts only and posts with replies. On people in the data, the first two options stay with chosen authors, while the last two bring in the public at large, with full threads reaching people furthest from the original post. On access, difficulty rises with depth and the gap between option 2 and option 3 is where many API tiers draw a line. On sync, counters need timed revisits, posts need deletion handling, replies need both plus orphan handling, and threads need periodic reconstruction. On analysis fit, counts answer attention, posts answer content, replies answer reception, threads answer structure.

A practical way to use this comparison is to write the research or business question first, find the shallowest option whose analysis-fit row answers it, and only then check the access and sync rows for that option. Going one level deeper than the question needs is the most expensive way to feel thorough. The table below condenses the comparison.

CriterionCounts onlyPosts onlyPosts with repliesFull threads
Volume per postOne itemOne itemGrows with popularityNo reliable ceiling
ContextAttention onlyWhat was saidHow it was receivedHow the conversation unfolded
People in the dataChosen authorsChosen authorsAuthors plus the publicWidest, including distant repliers
AccessEasiestEasyUneven by platformHardest
Sync burdenTimed revisitsDeletion handlingLate replies and orphansPeriodic reconstruction
AnswersReachContentReceptionStructure

If a provider such as ScrapingIsNotACrime is collecting on your behalf, specifying the depth in these terms, with the criteria written down, is clearer than asking for everything.

Which depth for which project

For brand or public-account monitoring, where the question is what accounts post and how much attention it draws, posts only with timed counters is the recommendation. It is cheap, stable under platform terms and keeps the dataset limited to accounts that post for an audience.

For customer-feedback or reception analysis, where the question is how people responded, posts with direct replies is the recommendation, with three conditions: a written justification for including repliers, a fixed capture window per post so samples are comparable, and a rule for orphaned and deleted replies. Collect reply text and the minimum identifier needed to deduplicate, and drop identifiers that the analysis does not need before the data leaves the collection system.

For academic studies of conversation dynamics, full threads is the only option that answers the question, and it should be treated as a project in its own right: researcher-program access where it exists, ethics review that addresses the people furthest from the original post, a sampling plan that caps thread size or picks threads deliberately, and a plan for how the data will be shared, following the options in sharing social media datasets.

For dashboards and trend tracking across many accounts, counts only is enough, and it is the only depth that scales to thousands of accounts without the request budget becoming the project. Whichever depth you pick, record the choice and the reason in the data specification, because the next analyst to open the dataset will assume the replies are either complete or absent, and both assumptions are wrong unless the document says so.

Share ๐• in f โœˆ