"And Jesus, walking
by the sea of Galilee,
saw two brethren,
Simon called Peter,
and Andrew his
brother, casting a
net into the sea: for
they were fishers.
And He saith unto
them, Follow me,
and I will make you
fishers of men. And
they straightway
left their nets, and followed him"
Matt 4:18-20

They went immediately (straightway) and left
their nets to follow Him. They had no idea what
their choice would mean or where it would take
them. They may have thought He was gathering
men to fight for a new physical kingdom by
revolting against Herod.

It isn't certain if they followed Jesus on all His
missions but Mark 1:39 seems to say that at
the beginning He was alone. It took some sort
of planning on the part of Andrew and Peter to
be able to leave their business and family for
long periods of time.