Kernel development

[goatedpenguin]

I intend to make a kernel, some of you may have seen that in other posts. However, what has struck me is why make a kernel when they are already good enough kernels out there? Sure I will learn something but why should I? I want to make a mainstream kernel that people can use(and I can get enough contributors to actually make into something)from scratch so I just wanted to hear gripes you guys may have for existing kernels that simply cannot be implemented(ie take too long, too much money or just not worth it or not possible). Thanks in advance

 

P.S: Please don't tell me it's not possible or it will take too long or any of that BS. Like I said I want to start small and than make it something sizable with other people's help. 

[ieleja]

More than few years ago, I compile kernel because distribution default don't include support for video capture card, that I have and want to use. I learn some basics and some tricks, but I don't learn something deep and important and stay far from that I can put back in kernel development.

 

But as Linux start as experiment of curios man, I recommend you just try, just get 6.8.9. or even 6.9-rc7 source and compile it, then try to boot it.

 

Good luck!

[dcgreen2k]

Making your own kernel is definitely possible, but making one that people actually use is more difficult because existing kernels like NT, Linux, and BSD have had extensive testing and security audits.

 

If you want to make a kernel from scratch, I highly recommend reading through tutorials and documentation on the OSDev wiki. It walks you through creating a bootloader in x86 assembly and then a simple kernel in C. Here's the first tutorial.

 

If at some point you want to base your kernel off of Linux, the Linux From Scratch project is a good start.

[igormp]

14 hours ago, goatedpenguin said:

However, what has struck me is why make a kernel when they are already good enough kernels out there? Sure I will learn something but why should I?

- Because other kernels lack something that you find interesting

- Because you want to learn

- Because you can

 

All of the above are valid reasons to do anything.

14 hours ago, goatedpenguin said:

I want to make a mainstream kernel that people can use(and I can get enough contributors to actually make into something)from scratch

DON'T do something with the expectation of people liking/using it or that you'll gather contributors. Do it because YOU want to do it and enjoy the process. If people happen to stumble upon it and decide to give a hand, awesome! But by no means expect this to happen any time soon, if at all, otherwise you'll end up disappointed and lacking in motivation.

 

Lots of people build their own OSes regularly, specially in universities where you usually have some kind of operating system class where you go through the likes of minix and whatnot (see OSDev as linked above), but only a few keep up with it, or turn it into something bigger.

[goatedpenguin]

8 hours ago, dcgreen2k said:

Making your own kernel is definitely possible, but making one that people actually use is more difficult because existing kernels like NT, Linux, and BSD have had extensive testing and security audits.

 

If you want to make a kernel from scratch, I highly recommend reading through tutorials and documentation on the OSDev wiki. It walks you through creating a bootloader in x86 assembly and then a simple kernel in C. Here's the first tutorial.

 

If at some point you want to base your kernel off of Linux, the Linux From Scratch project is a good start.

I intend to make the kernel from scratch and I have been digging some up some classes from MIT on how a kernel works before I can actually plant out on making one...
I probably however should try out LFS to really understand Linux as an OS/kernel since that is a goodstart  

I have been trying to learn C and I think asm is going to be needed one way or another if I want to make a my own kernel so I have been reading K&R to supplement that.  

 

6 hours ago, igormp said:

- Because other kernels lack something that you find interesting

- Because you want to learn

- Because you can

 

All of the above are valid reasons to do anything.

DON'T do something with the expectation of people liking/using it or that you'll gather contributors. Do it because YOU want to do it and enjoy the process. If people happen to stumble upon it and decide to give a hand, awesome! But by no means expect this to happen any time soon, if at all, otherwise you'll end up disappointed and lacking in motivation.

 

Lots of people build their own OSes regularly, specially in universities where you usually have some kind of operating system class where you go through the likes of minix and whatnot (see OSDev as linked above), but only a few keep up with it, or turn it into something bigger.

Thanks for the info I have been on OSdev and that site is a wealth of information. As for ppl disliking it/liking it is not my main goal but is something I want. Main reason I am making the kernel is I want to. 

[wasab]

Operating system is one of those theoretical stuffs you learn in school but rarely ever use in real life. I throw things like compiler and database relational theories in there as well. I would challenge anyone who works with database management system on a daily basis to define the word "Armstrong's axioms" or "relational algebra" for example. These are completely irrelevant for pratical purposes unless you are actively trying to create the next mysql/postgresql database. 

 

Only learn if you intellectually curious. You won't be working with any of these stuffs professionally unless you are a qualified PhD and got recruited by some R&D teams. 

[goatedpenguin]

13 minutes ago, wasab said:

Operating system is one of those theoretical stuffs you learn in school but rarely ever use in real life. I throw things like compiler and database relational theories in there as well. I would challenge anyone who works with database management system on a daily basis to define the word "Armstrong's axioms" or "relational algebra" for example. These are completely irrelevant for pratical purposes unless you are actively trying to create the next mysql/postgresql database. 

 

Only learn if you intellectually curious. You won't be working with any of these stuffs professionally unless you are a qualified PhD and got recruited by some R&D teams. 

I really am curious its not just for the sake of it plus I want to learn about it too. 

[Nayr438]

20 hours ago, goatedpenguin said:

I probably however should try out LFS to really understand Linux as an OS/kernel since that is a goodstart

As someone who has built LFS based on the Guide, it was the most pointless thing Ive ever done. The only thing you will really learn is how to cross compile. You will learn some basics about system management but not a lot and not anymore than you would Setting up Arch or Gentoo, in fact those would probably be more worthwhile. You can pick apart how packages are put together and create your own. In terms of learning how the kernel works with the system, you likely won't learn much from any of it. You would be better off digging through it's docs.

 

With that said as mentioned already OSDev is a great place to look at.

[igormp]

2 hours ago, wasab said:

Operating system is one of those theoretical stuffs you learn in school but rarely ever use in real life. I throw things like compiler and database relational theories in there as well. I would challenge anyone who works with database management system on a daily basis to define the word "Armstrong's axioms" or "relational algebra" for example. These are completely irrelevant for pratical purposes unless you are actively trying to create the next mysql/postgresql database. 

 

Only learn if you intellectually curious. You won't be working with any of these stuffs professionally unless you are a qualified PhD and got recruited by some R&D teams. 

That's a really superficial take. If all your work with is webdev building basic lading pages, then sure, but why did you get into university to begin with?

 

Anyhow, OP already mentioned that they want to make it for fun and to learn, that's more than enough reason.

[goatedpenguin]

42 minutes ago, Nayr438 said:

As someone who has built LFS based on the Guide, it was the most pointless thing Ive ever done. The only thing you will really learn is how to cross compile. You will learn some basics about system management but not a lot and not anymore than you would Setting up Arch or Gentoo, in fact those would probably be more worthwhile. You can pick apart how packages are put together and create your own. In terms of learning how the kernel works with the system, you likely won't learn much from any of it. You would be better off digging through it's docs.

 

With that said as mentioned already OSDev is a great place to look at.

Thanks for saving me a lot of time(months maybe) from doing LFS I sometimes wonder how you would learn it. But it is a fun challenge to take, you inevitably learn stuff even if it is not related. Leonardo da Vinci figured out how the human heart works from studying tornadoes, point if that I like to wander around becuase being hyper focused in one field is not always good. 

[goatedpenguin]

Alright to summarize, these are the resources I have planned out:

 

K&R
Osdev
The unix programming environment book

EDx

MIT opencourseware

Please recommend more if you know any thanks!  

[wasab]

1 hour ago, igormp said:

That's a really superficial take. If all your work with is webdev building basic lading pages, then sure, but why did you get into university to begin with?

well, there isnt one unless you wanna do this and you better have a resume that's more than I self-taught this off an online tutorial. 

2 hours ago, igormp said:

Anyhow, OP already mentioned that they want to make it for fun and to learn, that's more than enough reason.

problem with this approach is that this knowledge is just like learning to speak a language. if you don't use it, you will forget it. op can learn everything he wants to know, without it being part of his daily life however, everything will go flying out the window within a very short period. things like these are just muscles. you don't work them out regularly, they atrophy. 

[goatedpenguin]

6 minutes ago, wasab said:

well, there isnt one unless you wanna do this and you better have a resume that's more than I self-taught this off an online tutorial. 

problem with this approach is that this knowledge is just like learning to speak a language. if you don't use it, you will forget it. op can learn everything he wants to know, without it being part of his daily life however, everything will go flying out the window within a very short period. things like these are just muscles. you don't work them out regularly, they atrophy. 

I agree I wanna make this something that can grow.

I found these courses from mit:

 

https://ocw.mit.edu/courses/6-828-operating-system-engineering-fall-2012/download/

https://ocw.mit.edu/courses/6-858-computer-systems-security-fall-2014/download/

they lgtm but I just want your opinion, I really want to dedicate time to this. 

[wasab]

45 minutes ago, goatedpenguin said:

I agree I wanna make this something that can grow.

I found these courses from mit:

 

https://ocw.mit.edu/courses/6-828-operating-system-engineering-fall-2012/download/

https://ocw.mit.edu/courses/6-858-computer-systems-security-fall-2014/download/

they lgtm but I just want your opinion, I really want to dedicate time to this. 

This was the book I used in college, the class assignments were all hilariously done in Java on some toy Java simulator that mimics a kernel which all came from this book. The professor(he wrote the book) was atrocious and i barely learned anything but on the other hand, the class was an easy A with an easy introduction to the surface-level OS concepts without all the hardcore and tedious details that real kernel developments would be involved in. think of this like flight simulator vs actually piloting a Boeing 747. I would honestly just stop here as an amateur kernel developer. 

 

Carnegie Mellon has some of the more rigorous assignments on this if you want the actual hardcore stuffs with all the programming assignments done in C. 

 

edit: your 2nd link is on system security. completely different subject and topic.

 

edit 2: your first link also seems to have little to do with operating systems kernel. those are more like unix system fundamentals. maybe it was just introducing the subject by teaching some common system apis. i see it only has materials up to its first exam. 

 

i would try to explicitly search online for OPERATING SYSTEM. 

[goatedpenguin]

40 minutes ago, wasab said:

assignments were all hilariously done in Java

Lmao what

EDIT: Before I delve into making a kernel I want to first contribute to the linux kernel just to get a feel for kernel dev etc.

[wasab]

1 hour ago, goatedpenguin said:

Lmao what

i mean if you want to actually code up a kernel in C and assembly then by all means go for it. There were two professors at my school who all teach the same operating system course. first one plagiarized all of his teaching materials off of Carnegie Mellons and his operating system was downright hardcore as a result, all assignments done in C, 10 hours+ per week on homework and assignments at the very least.

 

2nd professor was the one i mentioned above. it was literally like 90% less work but still learning all the surface-level stuffs. most students picked the easy professor even if he sucked at teaching. 

 

1 hour ago, goatedpenguin said:

Lmao what

EDIT: Before I delve into making a kernel I want to first contribute to the linux kernel just to get a feel for kernel dev etc.

you should note Linux kernel maintainers have a very rigorous review process. all of your contributions and every line of code will be torn apart, carefully examined, and ruthlessly critiqued. You better have a thick skin because their reviewers are known to be harsh *cough Linus Torvalds *cough

 

Edit: anyways, I don't want to sound like I am discouraging you. This is some helpful beginner guide and resources to get your feet wet. Talks about the process to start debugging, making changes, test, send upstream, gaining experiences ect. 

Linux foundations has a lot more training videos and tutorials. 

 

[goatedpenguin]

1 hour ago, wasab said:

You better have a thick skin because their reviewers are known to be harsh *cough Linus Torvalds *cough

I have seen enough mailing lists from Torvalds and contributors to know how the culture is like there. That man is a piece of work yet one of the best. 

Thanks for all the info! 

[goatedpenguin]

Hopefully in sometime I will be able to make something and put it on github will share it with yall when available! @wasab @Nayr438 @Eigenvektor @igormp @dcgreen2k @ieleja

[goatedpenguin]

20 hours ago, wasab said:

edit: your 2nd link is on system security. completely different subject and topic.

 

I mean if I want people to actually use it and find it worthy to contribution I can't just be making unsafe code/programs that are vulnerable to buffer overflows, privilege escalations, RCEs etc. 

[goatedpenguin]

These are all the links I have found for OS and kernels:

 

Though I cannot find a link that explicitly explains me what a kernel is and how it works @wasab:

 

https://dokumen.pub/introduction-to-operating-system-design-and-implementation-the-osp-2-approach-97https://csd.cmu.edu/15410605-operating-system-design-and-implementation81846288425-9781846288432-1846288428.html

 

https://ocw.mit.edu/courses/6-858-computer-systems-security-fall-2014/download/

 

https://ocw.mit.edu/courses/6-828-operating-system-engineering-fall-2012/download/

 

https://wiki.osdev.org/Main_Page

 

https://colorcomputerarchive.com/repo/Documents/Books/The C Programming Language (Kernighan Ritchie).pdf

 

http://www.staroceans.org/kernel-and-driver/Linux System Programming--Talking Directly to Kernel %26 C Library.pdf

 

https://csd.cmu.edu/15410605-operating-system-design-and-implementation

My roadmap:

Using obsidian md to take all the notes

 

Learn C and asm first

 

Learn how a kernel works and its functions -> Need to find a source for that

 

Read Linux system programming book -> contribute to some drivers/stuff to get a feel

 

OSdev, OSP 2 book and os design and implementation @carnigemellon, trying to decide if I should do OS system design and OS system security from MIT opencourseware or not

 

Grind

 

 

Please help make this roadmap better but ik for sure that learning asm and C are crucial before starting anything else... 

 

 

[Gat Pelsinger]

On 5/9/2024 at 9:50 AM, goatedpenguin said:

why make a kernel when they are already good enough kernels out there?

Then don't.

 

If Linus Torvalds obeyed this, Linux won't be around that we know. But are you Torvalds 2.0?

[goatedpenguin]

5 hours ago, Gat Pelsinger said:

Then don't.

 

If Linus Torvalds obeyed this, Linux won't be around that we know. But are you Torvalds 2.0?

He had a reason to do it at that time and as I have stated, I intend to make one because I want to but I am just trying to figure on what stuff it should have that other kernels don’t have.

[Gat Pelsinger]

@goatedpenguin

 

You are saying like you are an experienced kernel developer and trying to start a start-up.

 

Even if your goal is to get into operating systems, competition today is very high and it probably won't be better than Linux.

 

Still though, there is no reason to not to try. A Linux alternative will still get attention. Don't you know what masterpiece Tarry Davis made? And you don't actually have to make a new kernel, you can just be a good contributor to the Linux kernel itself.

[Gat Pelsinger]

On 5/11/2024 at 9:56 AM, goatedpenguin said:

Hopefully in sometime I will be able to make something and put it on github will share it with yall when available! @wasab @Nayr438 @Eigenvektor @igormp @dcgreen2k @ieleja

Weren't you learning C a few months ago? Now you wanna develop your own kernel? Those are jokingly big steps.

 

Edit - Forgive me by what "sometime" means. That word really has a long range.

[Nayr438]

5 minutes ago, Gat Pelsinger said:

Weren't you learning C a few months ago? Now you wanna develop your own kernel? Those are jokingly big steps.

Not really. The best way to learn is to start actually working on something and watch it progress, doesn't matter what it is as long as you find enjoyment out of it.