How to explain that sample size does not influence project length

We have big enterprise projects they normally involve copying data from a source database to a destination database and then setting up a number of additional applications that sync this data etc.

The last project contained 250,000 items (rows of data). The next project will only contain 4,000 items. Project managers / business people believe the project should be 1/10 the time to complete because its only a fraction of the size of the last project.

What is a good analogy I can use to explain that writing code to transfer data from one system to another takes the same amount regardless of the number items – writing it for 1 item or for 100,000,000 will take roughly the same amount of time from a programming point of view.

Tell them it’s like building a new four lane highway to a remote part of the country. Whether that road gets used by 100 cars a day or 1000 cars a day, the effort to create the road will be about the same.

Granted, if it’s going to support 1,000,000 cars a day you’ll have to make the road a little more robust, but regardless, you’re going to have to cut down the same trees, blast through the same mountains, level the same amount of dirt, and these activities are pretty much a fixed cost no matter how many cars use the road.

Give them a calculator and ask them to add 1238783423 to 9858238483, time how long it takes. then ask them to add 3423 to 8483 and tell them you expect the answer approximately 100000 time quicker.

You might also explain the amount of data (probably) effects the length of time the software will take to run not the development time.

Put it into manager speak.

If you build a machine to make widgets at 1 widgets per second,
it does not matter if you use it to make 100 widgets
or 10000 widgets, the machine itself takes the same time to build.

the difference is at run time, not build time.

All management classes work on problem like this with hypothetical widget factories.

Don’t use an analogy. Just explain it.

For a very small number of items (10?) it’s cheapest to manually convert. Don’t write a program at all.
For a small number of items (100?) it will be worth writing a program. You may be able to make savings by ignoring some permutations of the data that are theoretically possible, but don’t appear in practise in the small dataset. Or appear in such small numbers that the program can reject them, and they can be converted manually. It’s feasible to run quick analyses on the data to check whether corner cases actually appear in the data. If they don’t appear, they can be ignored.
Once you pass this point, the actual size of the data has no impact. You need to write a serious program that can handle any possible input. The program can handle 1,000 items or 100,000. It just takes longer to run.

Education is better than talking down 🙂

Not really an analogy, but I still believe a good way to deal with this argument: demonstrate there is a fatal flaw in it.

Your previous project included (from what I get) copying data with some modifications on it.

If I got it right, that’s something a team of, say, 100 accountants can do in a matter of a few months. Then why did they throw software developers at the problem?

Because the software you created doesn’t care whether it will process 10 or 10 million pieces of data (not exactly, but I doubt your managers care for O(n) complexity). Thus, it was probably cheaper, faster and cleaner (less error-prone process).

If you’re more radical, you might even suggest that if they don’t like how fast the software team works, they can always call in the accountants to do the work by hand.

This made your managers’ lives much easier while you were developing the last project, and now, when they have to apply the same logic to figure out the next piece of software doesn’t care either if it’s going to work on 10 million or 4 000 rows, they suddenly forget about it.

I think in your case the managers are simply playing an estimation game and are trying to force the team to work faster, by pointing out the difference between 4000 and 250000 and hoping for some ‘guilt’. I could be wrong, but I’ve seen this done before.

It’s a terrible way to manage a team of programmers (actually any type of creative team) and it doesn’t help anyone.

I know you asked for an analogy, but I think that’s the wrong technique.

I believe, as others have mentioned in passing, that you need to emphasize that data size affects run time, not build time.
So, break it down for them – you actually have two sub-projects, building and running. The building project should (for the most part) be irrelevant of how much data it will run on, it only matters the types of data.
As for the runtime – sure, they can factor that according to data size (excluding any non-trivial fixed overhead).

It’s like if you have to drive to Melbourne – but first you have to build the car.
Sure, driving to Sydney might be faster – but building the vehicle takes the same amount of time.
_{Okay, I gave you an analogy after all.}

Maybe a telephone? Your customer wants a custom-made phone. If he makes 0 calls per day or 100 calls per day it would take the same amount of time to create his phone.

The data a phone transmits is analogous to the data copied by your program.

Your managers seem to confuse dev-time with the actual run-time of the program. But their misunderstanding may be different. They may assume that there are fewer “fields” involved. Not just fewer data records. If there are 100000 individual data fields it would be a massive dev effort compared to only 10 fields. More mapping work from system to system. In this case they may actually be correct, but there is still some constant overhead involved and you can’t simply divide by the number of fields to get the time.

As i like to describe it data has 2 dimensions length and width.
Length is the number of records, width is the total number of columns across all tables

Now when you want to import data it is like getting a block through a hole. You need to make a hole big enough for the smallest dimension, and then carry the block through

now with 10 million and 10 thousand the smallest dimension is still the width. So it is the width that decides how long it takes to make the hole.

TO complete the metaphor, ff it is the length that is smaller you would just type the data in manually

I import hundreds of client files every week.

One thing I have found is that the small files generally take longer to develop the data import for because:

They are less likely to follow the rules (we have standard file
structures, I have never seen a small client give us the data in the
standard format that we ask for but large ones understand why that is
important)
They tend to have more data integrity issues especially if they are
coming from an Excel file rather than a database (where the large
files tend to come from) which already had data integrity rules built
in
They are less likely to be provided in the same format every time.

We have found that we save a lot of time in development by building a parent child SSIS package that has a standard child process and any necessary manipulation to get the data in the form of the standard can be done in the parent. That way, it becomes less an issue of how many records when we do an estimate but an issue of how close to the stanrdard is the file we are getting. We now don’t get as many complaints when smaller things take longer to develop because they don’t fit the standard.

Writing a program is kind of like hiring a new employee. You have to teach them where to find the data, what to do with it, and how to give you the results. You have to keep an eye on them for a little while to make sure they’re doing it right. It might take a little longer to train them if they have a complicated/important job or if they’re going to do a very large amount of work, but it takes a substantial amount of time no matter what.

Many managers are familiar with the overhead involved in training a new employee, so this might make sense to them.

(the analogy breaks down insofar as your new employee is a superpowered robot who can get the job done in a trivial amount of time no matter how many records you throw at them, but hopefully you’ve made your point by then.)

Filed under: softwareengineering - @ 09:24

Thẻ: analogy

Thiết kế website giá rẻ

Danh mục

How to explain that sample size does not influence project length