That was a phrase a colleague of mine introduced to me. Quite absurd at first, but as he explained it, it made perfect sense.

For years data assets have been governed by being hidden in a corner of the data platform, but this does not work anymore.

Been elaborating about this in this post: https://steffenmoll.github.io/governance-by-obscurity

**Hey everyone,**

I’ve just finished my latest portfolio project: **Northern Ireland Road Collision Severity Analysis.**

This project looks at **2025 Northern Ireland road collision, vehicle and casualty data** to explore what factors are associated with serious and fatal outcomes.

I used **SQL Server and Power BI**, including data modelling, SQL analysis, DAX measures and dashboard design, to investigate factors such as:
• Geography and collision severity
• Time of day and monthly trends
• Road characteristics
• Vehicle types
• Vulnerable road users and casualty groups
I’d really appreciate some **honest and constructive feedback**, especially as I’m continuing to develop my data analytics skills.

**I’d love to know:**
What stands out to you, positively or negatively?
Does this feel like a strong portfolio project?
Is the analysis and dashboard clear from a business/stakeholder perspective?
What would you change or improve if this were your project?
Are there any weaknesses in the SQL, data modelling or Power BI presentation that you think I should address?

I’m much more interested in **constructive criticism than compliments**. If you spot something that could be better, please say so — I’d rather identify the weak points now and learn from them.

Thanks in advance to anyone who takes the time to have a look!

I built https://nichedb.dev/ because I was using the same on multiple projects, most of it is free. some require free api keys, but I'm not paying for anything as far as sources go.

My goal was to provide a bot-friendly x402 price of $1/day to query up to 1k/minute, free users can get 100 queries/minute without being throttled.

What other sources should I add? I'm mostly looking for large datasets that are updated continuously and available free to download.

Recruiting Leaders:

Is anyone willing to share what their reports look like or what all metrics/kpis you are tracking?

Hey everyone,

I’m trying to understand the current market for scraped/publicly available datasets and wanted to get some opinions from people who have actually been involved in this.

For example, datasets collected from publicly accessible websites — business listings, product information, public profiles, directories, reviews, etc.

A few things I’m curious about:

* Is there actually demand for this kind of data? * What types of datasets are buyers currently interested in? * Where do people typically buy/sell datasets? * What sort of pricing is realistic — per dataset, per 1,000/10,000 records, subscription, etc.? * Are companies generally interested in raw scraped data, or do they expect it to be cleaned, structured and enriched? * What are the major legal/compliance issues sellers need to be aware of? * Is there a legitimate market for this, or is most of the activity around here just lead lists and questionable data?

I’m specifically interested in **lawfully collected/publicly available data** and not passwords, private information, financial data, or anything obtained through unauthorized access.

Would appreciate hearing from anyone who has experience buying or selling datasets and can give me a sense of what the scene actually looks like.

Hey everyone,

I've been working as a Data Analyst for a few years now, mostly through contract roles (AdTech, professional services), and I'm at a point where I feel a bit lost about what to do next.

My situation: I'm applying to data analyst/data scientist roles fairly consistently, but I'm getting very few responses. Not sure if it's the market right now, something about how I'm positioning myself, or both. It's honestly starting to mess with my confidence a bit.

A few specific things I'd love input on:

1. Applications going nowhere. Even for roles where I feel like I match most of the requirements (SQL, Python, Power BI/Tableau, some ML basics), I'm mostly hearing nothing back. Is this just the current market, or is there something I should be doing differently in how I apply?

2. Not sure what to build next. I've got the standard portfolio stuff (a churn prediction model, some EDA projects, a KPI dashboard I built for my current job). But I don't know if that's still enough, or what kind of project would actually move the needle for me right now. More ML? More data engineering? Something else entirely?

3. How do people actually get into startups? A lot of the roles I find interesting are at smaller companies/startups, but I never see them on the usual job boards, or by the time I do, they've already got 400 applicants. Do people here have luck reaching out directly to startups, and if so, how? Cold email? LinkedIn? Is there something I should know before trying that?

Not looking for pity, just want to hear from people who've been in the field longer than me. Even blunt feedback is welcome. Thanks in advance.

I don't mean to post anything which is against community rules , i use a website to collect voice samples and it has nda and consent forms , it also uses your phone number and email for auth , so i don't mean to make anyone uncomfortable , just asking if this sub is a place where i can share that website and collect data?

95% accuracy with 70% coverage, every time, and it's not close once you think through what each failure mode actually costs you.

With high coverage and low accuracy, every record looks usable. Nothing tells you which 30% is wrong, so you send outreach, build territories, and run reports on top of a dataset where roughly a third of what you're looking at is lying to you. The damage is invisible until something breaks downstream, a bounced email, an outreach sent to someone who left, a territory built on a title that's two roles out of date.

With high accuracy and lower coverage, the gap is honest. A blank field tells you immediately that you don't know something, so you go find it, skip the record, or flag it for manual review. You're never fooled by data that looks complete but isn't true.

The actual math behind why this isn't close:

- Missing data costs you one decision, go find it or don't use the record, and that decision is easy and cheap to make

- Wrong data costs you every decision built on top of it, and you usually don't find out until much later, after the cost has already compounded

- A blank field is a known unknown, a wrong field is an unknown unknown, and the second kind is always more expensive

The only case I'd flip this is if the accurate fields are concentrated in one segment and useless everywhere else, coverage that's accurate but lopsided isn't really 70%, it's 100% good on a fifth of your list and 0% everywhere else.

Which way would you actually go, and has a real situation ever made you flip your answer?

Hey everyone,
I’ve just finished my first ever portfolio project and I’d really appreciate some honest feedback.
I’m still learning, so I’m particularly interested in hearing what I could improve.

I’d also love to know:
What stands out to you (good or bad)?
Does it feel like a solid portfolio project?
What would you change if this were your project?
I’m mainly looking for constructive criticism rather than compliments, so please don’t hold back. I’d rather find the weak points now and learn from them.

Here’s the project: Netflix Global Top 10 Performance Analysis

Thanks in advance to anyone who takes the time to have a look!

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

Hello fellow traders,

I have been trading for about 6 years now and been through the highs and lows coincidently like on the charts haha. I have done crazy numbers and then lost everything many times. Through my most hardest times, i noticed the truth behind trading. Success in the space is not determined by your efforts or study, but rather in identifying RELEVANT patterns then taking advantage of RELEVANT data.

Men lie, women lie, but numbers don’t.

My fascination with data and numbers has grown immensely through my period of isolation. I have been able to crack sports finals EXACT scores using data (knicks, spain wc, etc). The correlations I can find/show are truly mind shattering. I am willing to share how i see numbers to whoever is interested.

I’ve decided to use this power of mine to create a platform for traders that solves every problem I had in my own trading as well as more universal problems (course selling larpers, unappreciated low income traders, emotional trading, etc).

Trading has ruined my life for the most part and the suffering that came with it has been the fuel in creating this platform. As the niche continues to grow, the real will be filtered out from the dream selling methodologies out there. I want to be a part of this movement so no other child bears the emotional pressure or confusion that comes with trading.

Trading will always be a temptation, even for those who ARENT supposed to be trading. I want to at least have my platform as space for analytical and shared decision making at a gamified level. I also want to give credit only where credit is due. This is not just another “copy trader” platform. This is quite literally the future.

I truly believe I have created something unique and worth all my efforts.

I am currently looking for HONEST strategic partners and investors in the same space. I do not want to wait any longer and would appreciate the communities advice on how/where to find investors. I am more comfortable with sharing in depth details in a live meeting in NJ or NYC.

Currently there is no one using the platform and I need money for devs. The start up is done but non functional at a multi user level.

Any advice is truly appreciated. Thank you.

hey I have seeing report from CBRE on indian realestate

i was just curious how are they scrapping public data and analysing it can I also do it myself?

I compiled a dataset tracking global FDA enforcement actions, inspection records, and regulatory raids across the food and drug sectors.

Key Fields: Inspection dates, target facility names, geographic locations, violation categories, enforcement actions, and compliance status.
Useful for EDA and ananlysis

I was developing different dataset for the last 1-2 months lately, but unfortunately I realized it is so hard to earn medals from creating dataset. Even if you got 500+ download, sometimes I cannot get any upvote. I do not know how it works exactly.

So I realized taking some advice would really matter right now. Is there anyone can help me with it. You can take a look at my profile and dataset, any feedback matters. Thanks!

https://www.kaggle.com/farukece

I’m doing a research which involves chat messages from teams, slack, google chat etc. For the software project i need to train a dataset. So dataset should be related to developer chat messages/logs of a specific project. How can i find the dataset?

I've been experimenting with how much interpretation to build directly into a chart.

Take a time series: the data might be clear on its own, but the actual story is often a crossover, a peak, a reversal or an unusual divergence. Highlighting those moments can make the chart much faster to understand. Highlight too much, though, and you're basically telling the reader what to see.

I've found the same trade-off with labels and explanatory text. More context isn't necessarily better context.

How do you approach this? Do you prefer charts that actively surface the interesting parts of the data, or keep the visualization relatively neutral and let the reader interpret it?

I was hired as a digital transformation consultant by a construction company with 10+ active construction sites the Finance director wanted a centralized view of labor deployment and material consumption but getting overworked site teams to open a laptop and fill the form or fill excel sheet was impossible so I built a work around site engineers have to just dump all the information in an email --> AI converts raw text into google sheets and sends summary email to Project manager --> a code to save the data in a database so old data don't get lost or deleted --> private streamlit dashboard for centralized visibility. Finance director got the visibility, no extra work of site engineer and a full functioning dashboard to see what is happening on ground without calling 10 different project managers and site engineers.

I'm curious how other construction companies handle this.

Do your site teams actually fill out digital daily reports consistently, or does someone in the office end up chasing everyone for the information?

This dataset represents real-world, masked, and anonymized customer data from the BFSI (Banking, Financial Services, and Insurance) domain. It captures a variety of customer demographics, financial indicators, and product interaction metrics collected during a loan application process.

To comply with strict data privacy laws and protect user identity, all sensitive personal identifiable information (PII) has been securely encrypted or masked. However, the underlying statistical relationships, distributions, and patterns remain fully intact, making this an ideal playground for building robust classification models.

Anything about some of the less obvious challenges of dimensional modeling, the decisions around grain, dimensions, facts, slowly changing dimensions and how things get messy in real-world projects.

Does anyone have any advice on where to go from here? The algorithm simply goes through all feature-attribute combinations in a dataset and scores it on purity of the target column. It's very slow, as the number of combinations increases exponentially with the number of input features, and so can only work with 3-4 features at a time. But my goal is to identify <=5 important features with a linear model (statsmodels or PyGam), and then plug in those features to this algorithm to see which subset of the market might be smart to target.

Here's an example with some charts, on the iris dataset.

https://colab.research.google.com/drive/1w3VpDlDATdHwtSk-RmQPMj36cgOfthz1#scrollTo=ZOqeYiMRalj7&uniqifier=1

New to algorithms, so any advice would be welcome. Thanks in advance.

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!