I started with a Kawai keyboard my mom bought on a shopping channel when i was a kid. I played guitar too, so I sent it through my pedals and such. I had a 12" tom that I found at a backyard gig. I used my walkman headphones as a microphone. At Radio Shack I bought a bunch of cables: Y-cables, 1/8th" to 1/4", RCA to 1/8th", etc. I'd record bits of records (anything we had in the house: Front 242 and Chameleons, to Doris Day and Wagner), my keyboard and my headphones and my drum, live into this badass Aiwa walkman I had. A bit later i procured a 4 track from a music store, raising my capabilities greatly. With mute buttons and faders, I was doing a lot of noise and collage, like Nurse With Wound, Coil, etc. A bit after that I got Cakewalk Professional 5 and an AWE 64 Gold; now, i could edit samples and trigger them with the computer keyboard.
To start making noise, It's really helpful to get to know a piece of hardware that interests you. Find a synth, or drum machine, or a piece of audio software, and learn it and know it like a favourite book. Speaking for myself of course, being curious about how things work, how sounds are made and what I can make with them is what naturally drew me into music, thus making music.
ACID's a good prog, and theres a free vesion, and im sure there is still a free veersion of Cakewalk. Of course, you can get an entry level piece of kit, like a 2 channel audio and midi breakout box, which will come bundled with all sorts of rad shit, like Cubase or Pro
Tools or Performer, etc.