How Often Do Databases Break?

[Hurican7]

The Problem:

I'm not a software guy. However, I do have some C++ and a growing amount of VBA/SolidWorks API macro experience. The small fabrication company I am currently working for needs to utilize some sort of database to organize its engineering/production files.

 

A Solution:

My current plan is to build a partially automated system (nothing too crazy - I don't plan on building a full bulletproof PoductDataManagement/ProductLifetimeManagement system) that uses a database with an entry for each file with its cross reference information for searches, as well as a link to open that file from the database. The longer we have observed the problems with the manual filing systems we want to replace, the more we have realized that our database solution is similar to a library database.

 

Another Problem:

However, some of the coworkers are worried about the reliability of a database driven system. Assuming we don't build something beyond our skill level, is there actually any reason to think that a database (even a relatively basic Microsoft Access database) would randomly crap out? They don't seem to realize that the world runs on databases and they generally have greatly reduced error rates than manual systems - granted they are well made and maintained according to their complexity. 

 

In short, I need reasons to easily convince them and management - or reasons why I'm on the wrong path. Your help is appreciated. I will try to check this thread at least once or twice a day this week if you have suggestions.

 

Cheers!

[LtStaffel]

No reason something like that would, on its own, flunk out. If you keep backups and you aren't running something like

DROP DATABASE *

then I see no reason why you would have problems. You could (bluntly and offensively) say it like this: "The burden of proof is on you as to why this wouldn't work, seeing that millions of companies already use database software without major issues."

[Cakeman]

You're going to receive pushback from them on implementing a database system it sounds like. I would suggest showing samples of databases and how they are used in basically every company. 

 

I would suggest doing a phased roll-out with a individual hands on approach with each person in the company. Like, actual one on one time with each person who will be using this with a setup of the old version and the new version running next to each other and how they both accomplish the same and new tasks. 

 

I would also try to show your management how much more effective a new database system would be and a rough guess on how many hours it will save per year or something like that. Many tech illiterate managers don't really understand how new things work and are scared of change. 

 

The one big point in this though is that you will NEED to make sure the database has backups and cannot destroy the company if it isn't properly backed up in any way and a massive failure happens. From what it sounds like, your managers are scared of a situation like that and rightfully so. 

 

I'd also suggest maybe contacting other employees from other companies through Linkedin or something and asking how they tackle that issue. 

[vorticalbox]

2 hours ago, Cakeman said:

'd also suggest maybe contacting other employees from other companies through Linkedin or something and asking how they tackle that issue. 

How we tackle it is to use a company to manage the database and back ups for us.

 

We use mlabs for our mongo db and for heroku for our database, though we have moved most of our apps from heroku to aws fargate.

 

I build financial software for a number of large UK businesses.

[Cruorzy]

Been using a mysql database on daily bases with several tasks and relations for over 5 years and never broke down.

Always make sure to backup your database.

 

the wuestion is what is their solution? sounds like we want a new excel document statement.

[colonel_mortis]

The LTT forum database has been in constant heavy use for the past 5 years, and we have had one database failure in that time (and that failure resulted in a few hours of data loss on a subset of the database). That failure is definitely an exception rather than something that would have been expected, and although we were never able to find the root cause I suspect it was a hardware fault, but failures are possible. Your database would not be subject to nearly the stress that our DB is subject to, and with a reasonable backup strategy you would mitigate almost all of the problems that could occur if a problem did occur.

 

Most database software, such as MySQL or PostgreSQL, offer ACID guarantees. The D in ACID means that once you make a change, it is guaranteed to persist regardless of server power failure (though it may involve a recovery process prior to normal service resuming), so database software offers stronger guarantees than most programs do about data corruption. Think about how often you experience file corruption in your everyday computer use, and that will give an upper bound for how often a decent database program should fail.

[Mira Yurizaki]

Before you start looking into implementing some database, ask yourself what are the requirements of your organization and how will this data be normalized? For example, a document style database like MongoDB is an awful choice for a social network because MongoDB isn't very good for handling multiple relationships between data. A graph database is a much better fit.

 

Even if you choose something reliable, if you choose the wrong type of database, it'll make not only getting the system up and running hard, but maintaining it even harder.

[vorticalbox]

13 hours ago, M.Yurizaki said:

For example, a document style database like MongoDB is an awful choice for a social network because MongoDB isn't very good for handling multiple relationships between data

Mongoose in node has .populate that allows you take an array of ids and populate that array with the documents.

 

Admittedly not a nice solution but if you're using MEAN stack the  it's pretty good.

[Hurican7]

Sorry for not getting back earlier. I did read the replies yesterday.

 

Cakeman, I like your idea to do a phased roll-out per user; this is easily achievable since this is a small company. I started to do a side-by-side system analysis to find time/cost benefits but forgot when urgent projects were put on my desk. One of the employees here is taking graduate classes in data related stuff and mentioned one of those classes is about how to present data in a way that engages nontechnical users. I should be able to get a few pointers on how to do that since there is supposed to be a meeting about this stuff within the next month that I will most likely be involved in (whatever that means :P). 

 

colonel_mortis, that is a helpful use case. This DB won't be stressed nearly as hard as this forum's, but if we go through with it it should be resilient nonetheless since it will be dealing with job-critical information.

 

M.Yurizaki, while I think our options are limited due to budget/licensing constraints, I will definitely do more research into the correct DB to implement since this one will rely on relationships.

 

Rest assured, IT here has backups taken care of. Since this is a recurring of concern of most of the replies so far, I will double check with IT tomorrow but I am already confident in the backup solution from previous conversations.

 

Lastly, on a separate but related topic, I finished the first SolidWorks API macro (written in VBA) tool that will be incorporated into the database system. I will be developing a few more in the meantime, but each oine will operate independently of each other and independently of the database until that solution is figured out. This way, even if management goes full derp and decides against the development/implementation of a database, at least future engineers will be able to save a lot of time by automating some of the repetitive documentation procedures. A few of the VBA tools will probably have to be combined for integration into the database system or even interact with a script, but those complications haven't presented themselves yet.

[geo3]

On 6/11/2018 at 8:34 PM, Hurican7 said:

(even a relatively basic Microsoft Access database)

In my experience, these break more often than they work. A real database however is very reliable if set up properly.

[Mira Yurizaki]

53 minutes ago, Hurican7 said:

M.Yurizaki, while I think our options are limited due to budget/licensing constraints, I will definitely do more research into the correct DB to implement since this one will rely on relationships.

I think a good approach to figuring out what database you need is to write out the data you think the company needs to store, how it will all connect together, and the kinds of queries you think you'll be doing.

[Hurican7]

14 hours ago, M.Yurizaki said:

I think a good approach to figuring out what database you need is to write out the data you think the company needs to store, how it will all connect together, and the kinds of queries you think you'll be doing.

Exactly - thank you for reminding me of this. I have drawn up some of the functionality but not all of it. When I approach a programming challenge I usually write the process out by hand so I can find intrinsic flaws in the overall logic. It only makes sense that this method should be applied to DB choice and development.

 

 

15 hours ago, geo3 said:

In my experience, these break more often than they work. A real database however is very reliable if set up properly.

From my limited understanding, MS Access is a dumbed down database system designed to appear more user friendly for Excel users. Since the company does have a Windows server, it looks like SQL Server Express might be a good alternative to look into. I need to verify if it will be a good fit for the data types and relationships, as well as log keeping and some sort of user-friendly GUI to make the queries.

 

Just to note: we did try a basic Access database that worked better than the alternative (which is literally almost nothing - max LoLz) and even made a basic GUI for it but certain individuals were so turned off by it in its GUI-less state that they avoided it (to the detriment of the workflows involved). The way the Access DB was set up does have some intrinsic flaws but we never had stability issues in the last two years, at least to my knowledge.