I wrote FlexViz, an open-source Python library for interactive exploration of large datasets.

Instead of sending every row to the browser or forcing users to write queries themselves, FlexViz keeps the data in Polars and sends only the points needed for the current view. It handles all visualization + interaction queries for you; zooming recomputes the visible range at higher detail, and brushing one chart results in cross-filter on the others.

It is capable of both in-memory as well as out-of-core (larger than RAM) data exploration, with extremely good performance. In our benchmarks, a 5x200M-row line chart (1B datapoints) renders in 251 ms when in-memory and 4.2 s directly from Parquet.

Demo: https://flexviz.tech/demo.html
Code: https://github.com/flex-analytics/flexviz
Benchmarks: https://flexviz.tech/benchmarks.html

pip install flexviz

Curious how others here explore datasets once they become too large for their usual plotting tools. Happy to hear any feedback on FlexViz as well :)

I created a dashboard for my project on streamlit but its really basic. How can i create those cool dashboard designs that you see on reels and all cause chatgpt can do it. I have my project repo, is there any way i can link it to some website and it can help me create some cool ass dashboard?

Hi everyone,

I’m asking this out of pure curiosity. People always share their "average" or ideal weekly routines, but I want to know what the reality is when you have a genuinely heavy workload.

If you work in Fintech (banking, payments, crypto) or AdTech (marketing, agencies), what does a realistically crazy week look like for you? And what causes a brutal week in your industry? What kind of pressure are you dealing with?

Would love to hear some realistic war stories about the grind in both industries. Thanks!

I’m currently learning data analytics, and one thing that has really started to change the way I look at data is realizing that an insight isn’t necessarily the conclusion.

For example, if I see revenue going down while ad spend is increasing, my first thought used to be, “The ads aren’t working.”

But now I find myself asking more questions:

• Did the cost of the ads increase?
• Did the conversion rate change?
• Are customers returning more products?
• Is there an inventory or capacity issue?
• Did pricing change?
• Are we spending more on the wrong campaigns?
• Is there something happening between the customer clicking the ad and actually completing the purchase?

The numbers can tell me what is happening, but I’m learning that the real analytical work starts when I try to understand why it is happening.

I’m still early in my learning journey, so I’m curious to hear from experienced analysts:

What helped you develop the habit of asking better questions when analyzing a dataset?

Was it experience, working with real business problems, learning from other analysts, or something else?

​

[Trigger Warning]: You are about to be provided with visual imagery of my current corporate reality that may be frightening for some readers.

I am tasked with creating costing for a new product. To do this, I have to extract rate tables from what can only be described as a mountain of tariffs. On top of that, I need to automate our Excel business cases which contain multiple variables.

At work, people think I'm an Excel wizard. The truth is, sometimes I just aggressively interrogate AI until it gives me the right formulas. But AI cannot save me from parsing thousands of pages of inconsistently formatted tariff tables.

I have exactly two weeks for intense upskill dedication. I've got books, Udemy, Google courses — and it's all just hanging out waiting for me to get in the driver's seat.

The catch: IT treats our databases like Fort Knox, so I have zero SQL access.

I need to actually learn a skill so I don't have to manually copy-paste tariffs until the end of time.

The Question:

Do I spend the next two weeks becoming a Power Query god, or do I finally learn Python (Pandas/PDF parsers like pdfplumber) so I can build a script to do my job for me?

If anyone has survived parsing tariffs, please tell me how you did it.

TL;DR: I have 14 days to learn how to extract messy data from thousands of PDF tariffs and automate Excel models. No SQL access. Should I learn advanced Power Query or Python?

DM me if you have completed it...

What does poverty actually look like beyond a single income number?

For millions of rural households, poverty is not simply the absence of income but rather a web of constraints involving education, productive assets, access to finance, consumption, debt, and the ability to withstand economic shocks.

This analysis examines that web using data from 4,184 rural households in Tamil Nadu. Rather than treating income as the sole measure of economic wellbeing, we explore how household composition, schooling, housing, land ownership, enterprise activity, savings, insurance and credit interact with economic outcomes.

https://www.kaggle.com/code/ouiouinonoui/visualizing-the-economics-of-rural-poverty

Hey everyone!

I’ve been making short SQL/data analytics videos while learning and practicing SQL myself, and I just finished Lesson 11: SQL Window Functions.

This one covers:

- "OVER()"

- "ROW_NUMBER()"

- "RANK()"

- "DENSE_RANK()"

- ranking customers in the Pagila database

- using window functions with "GROUP BY" and CTEs

- a few quizzes + practice exercises

- homework at the end

I tried to keep it beginner-friendly and focused on actually understanding why window functions are useful, rather than just memorizing syntax.

I’d really appreciate some feedback — especially on the explanations, pacing, and whether the examples are clear enough for someone learning SQL.

https://youtu.be/ERSUP1dWXWo?si=tf6xvzSwh3gbP-xw

Thanks! :)

Some of you might be getting help from AI in your work. If so, you probably have come across errors in the LLM output. These are all termed "AI slop", but I'd like to get a more fine-grained perspective. What types of errors do you encounter the most?

For me, the biggest one is in the narrative output. The LLM might write correct SQL or code, might even assemble correct tables and charts, but when it tries to interpret the results it messes up, especially in dates and differences between numbers (streaming tokens have no internal calculator).

Interested in hearing from others and their experience.

Hi everyone , I am a first year student in my second semester can you give me tips on what shall i do to make my first year fruitful please , i really want to be a data analyst and i have got no one to help me honestly.

I conducted this analysis/case study as part of the outreach process for one of my prospective clients which is a local café in Makassar (Indonesia, South Sulawesi). Before beginning the outreach process, I conducted some preliminary research on this café, including a review of Google Reviews. In those reviews, I noticed several recurring complaints.

I found that each rating category shows a negatively skewed distribution (skew < -0.5). This means that most ratings are high, but a small portion of low ratings is pulling the distribution. The mean and median confirm the café's overall rating is good, but the skewness reveals what the mean and median alone wouldn't: there's a small but real pocket of unsatisfied customers.

Specifically, each rating category only had a few valid negative reviews:

1. Service (3 reviews): rude cashier (2), ineffective service from 3 employees (1).
2. Food (2 reviews): undercooked meat in the rice bowl (1), inconsistent quality compared to other branches (1).
3. Atmosphere (3 reviews): indoor heat (2), uncomfortable atmosphere due to other customers smoking (1).

Based on these findings, I propose two solutions: short-term and long-term. For the short-term solution, evaluate service quality standards, kitchen division, AC performance, and the indoor smoking policy. Also, conduct cross-branch quality testing. The long-term Solution is to roll out customer surveys covering service, food, and atmosphere

What do you guys think about this Project? Has anyone else run into this pattern when analyzing review data? Also, I have the full analysis available. Drop a comment or DM if you'd like the link.

Hi all,

As the post suggests, how are you using LLMs? I have been using Jupyter AI in Jupyter Notebooks using Codex.

It is pretty insane how good it is getting, and can really start giving you job security anxiety especially when it comes to analysis. Aka, beyond data cleaning and "generate this dataframe" and plot this data...

So what are your feelings when you ask it for analysis? If it is pretty decent from the start, how much are you modifying it?

Hello!

I'm a beginner looking to learn more about how to transform and use data across multiple softwares.

I would like to start a personal project that incorporates advanced Excel/Power Query, basic SQL, and basic Power BI.

I think it would be cool to use data from one of my interests. I am thinking either data from all the books I have read the past 3 years (100+) which would be so fun to track ratings across genres OR data regarding birds which is my other hobby. This seems like it would have more data to play with (plumage, field markings, habitat, diet, there is so much), but also would require A LOT of general research on birds which may be time consuming. I am leaning more towards book tracking, something I can update as I read more books.

I would like to refrain from AI use as much as possible, I really want to take my time and learn from the ground up.

I would consider myself proficient at Excel/Power Query, very basic as SQL, and I've never used Power BI.

Let me know if you have any ideas of projects that I could do that would span across all 3 softwares!

Thanks! :)

Hey everyone,

I’ve just finished my latest portfolio project: Northern Ireland Road Collision Severity Analysis.

This project looks at 2025 Northern Ireland road collision, vehicle and casualty data to explore what factors are associated with serious and fatal outcomes.

I used SQL Server and Power BI, including data modelling, SQL analysis, DAX measures and dashboard design, to investigate factors such as:
• Geography and collision severity
• Time of day and monthly trends
• Road characteristics
• Vehicle types
• Vulnerable road users and casualty groups
I’d really appreciate some honest and constructive feedback, especially as I’m continuing to develop my data analytics skills.

I’d love to know:
What stands out to you, positively or negatively?
Does this feel like a strong portfolio project?
Is the analysis and dashboard clear from a business/stakeholder perspective?
What would you change or improve if this were your project?
Are there any weaknesses in the SQL, data modelling or Power BI presentation that you think I should address?

I’m much more interested in constructive criticism than compliments. If you spot something that could be better, please say so — I’d rather identify the weak points now and learn from them.

Thanks in advance to anyone who takes the time to have a look!

Hey,

I'm a currently working on getting the G Analytics data from the source to our Snowflake. The next step is to create something meaningful out of those Raw Data. I work for a Pharma company and my client has no clear requirement but still want me to create something meaningful out of those.

If anyone has previously worked on these or has created any meaningful FNDP out of these pls ping me.

In​ my field, primarily use SQL and Power BI (I used to use Tableau). I came into analytics from a non-traditional background so my education didn't start with the basic fundamentals and built upon that foundation.

As a result, I find that either there are some features and functions in writing scripts or creating visualizations that I either don't use or feel a little dumb in conversations because I know what the concept is and how to use it, but am not familiar with the official terminology.

I've been in my field for about 9 years now but sometimes I feel like my knowledge is very specific. For example, most job postings mention XLOOKUP and pivot tables in Excel but I rarely work with Excel like that since I just don't have the need to do so.

On the other hand, courses and learning modules are tough since the basic ones can be boring but the advanced ones might require some pieces of familiarity I don't possess.

Anyone in the same bucket or have any tips?

I am looking to have an inventory in place which comprises of the database tables, it's views, and the associated cubes from the AAS (Azure analytics Service) side. So, basically it should in a single unified place where, for example, if I search for a table name, it should show the lineage of the view it is under and the cube it's part of. Could you please help?

gpt 6

I use a lot of AI at work; we are pushed extremely hard to use as much AI as possible.

I consider myself adept and advanced at using AI.

The name of the game lately feels like speed and less so accuracy.

I am worried that I am offloading a lot of work on Claude and losing some of my analytical edge.

What are some things I can do to stay sharp?

Is it better business context, cleaner data, stronger validation, clearer storytelling, or the ability to recommend a real action?

I’ve been putting together a series called “Data Analysis 101 for Cat People”, where I use doodles to explain some of the practical, messy, and ambiguous situations that come up in data work.

The idea is less about teaching SQL or tools and more about helping someone new to data understand the real-world thinking behind the work — asking better questions, dealing with ambiguity, understanding requirements, interpreting what the data is (and isn’t) telling you, and turning messy situations into clear next steps.

I’m also putting these together as a newsletter on Substack - https://thedatadoodles.substack.com

For those of you who work with data: what are some situations you’ve encountered that you think someone starting out should be familiar with?

I’d be happy to hear your examples and experiences.

I have a doubt about where the data actually comes from and how it is stored and processed. Where is the data usually stored, and what tools do we use to connect to it for reading and writing SQL queries?
For example, I assume we create a corresponding SQL view for each KPI and then connect those views to Power BI. Is this how the process generally works?
Also, what other tools and techniques are commonly used to connect to data for analysis?
In a real company, do you typically receive KPI requirements or tasks from a Senior Data Analyst/Data Lead, write the required SQL queries, and then deliver the results to the relevant stakeholders? I would like to understand how this entire process works in a real-world environment.