Should I store log files in version control

The log files in question are metrics log files. One of them is a project file produced by SourceMonitor, to which I regularly add checkpoints to chart the progress of projects.

Nowadays storage is cheap. If these are metrics that you can use to show performance increases that coincide with versions, then I’d say keeping them in source control is just fine. Just make sure that you’re storing them for a reason. If you can regenerate these logs/performance files from the source (meaning I can check out the code, run it, and produce the exact file you’re about to store), then you probably don’t need to.

Lots of things are stored in version control these days, including any and all documentation and other files that change along with the version of the project/code.

Note: You would store these in a branch outside your code branch. Like Thomas’ answer states, the source code branch that is used to build/deploy the app should be be separate so that your build server (or you, if you don’t have time/space) can download/checkout the code independently of the documentation.

I wouldn’t store them in version control, at least in the same project as the source code for the deliverable product. To me, it just doesn’t make much sense to combine what’s effectively logging data in a directory structure for a deliverable product. The repository used for the project shouldn’t contain anything outside of what is needed to build and deploy/deliver the system at hand (including instructions for building and deploying).

I would recommend storing them in the same place or using the same methodology as you do your other documents, such as test result history, design documents, requirements specifications, and so on. I don’t know what technology you use, but uploading them to a SharePoint server would work, as would uploading them to a shared data server and putting the path to them (and perhaps summaries of them) on a wiki page.

I should not add log file in version control, they are temporary and static so you don’t have to maintain the version of it. You also collect the results of the log files into a chart, so I don’t think it’s necessary to add the log files into version control.

IMO, no, it’s place in archived backups, if you wish to have that data later. General rule I use, store only things that you make youself or some others, if you say, include source code of other tool. Any files produced by codegen tools, logs, any other output data should be left out of repository, or stored separetly. The point is that, usually, you can checkout required version and recreate that data at any moment. The are exceptions, I suppose, but I have not encountered them so far.

Get your build server to gather your metrics and store them there against each build.

There should be some stuff that you only need the latest version of and some stuff that you need permanently to graph improvements. For example, if you build a whole code coverage report every build, you only ever need to look at the latest report to decide where to focus your test-writing efforts. But it is useful to keep code coverage %, branch coverage %, etc for every build and graph it out.

Jenkins is very good for managing this kind of thing. The HTML publisher plugin can be made to publish temporary, current reports (or you can keep them for n days, if you have the space and need), while the plot plugin can be made to track long-term progress.

You can also store them in a NoSQL database, then you could write an interface to get data reporting. I think NoSQL should be suitable for this.

Are they files that change over time?
Is it helpful to be able to retrieve older versions?
Would losing these files be a problem?

If you answered ‘yes’ to one or more of the above, then sure, go ahead and store them in a version control system. Since you answered ‘yes’, there’s obviously some benefit to be gained by doing so, and little reason not to.

The best argument I can think of against using version control for these files is that log files tend to be linear in nature — the only way you normally modify them is to append new data to the end of the file. Given that, they have a sort of primitive change tracking mechanism built in, so the added complexity of using version control may not be necessary.

Overview

It’s a good idea to back up log files, but they do NOT belong in source control!

Good Places For Backups

Burn them encrypted to DVD once a month and pop the DVD in a safe deposit box off-site.
In zip files on a secure drive in a separate secure server room.
Maybe in a secure NoSQL database?

Security

Log files often contain confidential information. Users can paste their password into the wrong field and have it end up in the logs. All kinds of information ends up there that’s probably covered by Safe Harbor and other international privacy regulations even if it’s not otherwise considered private by your customers. You want to limit the number of people who have access to this stuff and generally all programmers have access to source control.

Size/Speed

Source control is designed for little 100K-5M source files. Not Gigabytes of logging junk. Depending on the source control tool you use, you may find yourself waiting for weeks just to get the latest version of your code if you store huge logs in there.

Features

Source control is designed to compare versions. There is no version comparison like this with log files. If you need to reference these files quickly and constantly, then log table in a database (either SQL or NoSQL would work) is the way to go. Version control is just not a good feature-fit for this purpose.

Tools

Gnupg is great for compressing and encrypting huge things. Use this on the files on the server where the files are generated. They will transfer faster and be secure when they travel over the wire.
Linux/bash has a logrotate tool, but most logging tools have a logrotate feature that can keep sixty 100Meg files in your log folder just by renaming them .1, .2, .3, etc. Check how big a file your favorite editor can handle and keep each file a little smaller than that. Presumably your server drive is backed up and these files with it, but when you upgrade servers (e.g. for an OS upgrade) remember to put a copy of these files somewhere else and restore them to the new server when you are done!

Filed under: softwareengineering - @ 08:12

Thẻ: version-control

Thiết kế website giá rẻ

Danh mục