You can give it a shot to see what happens. I would absolutely, positively, for sure, start small and fence it in to keep out the groundhogs, rabbits & deer because they will clean you out. You will be amazed at how much you can get out of a 25 X 50 ft garden. If you get rain soon after you plant, you might see more weeds than crops when you return.
There are ways to overcome most of the challanges you will face, for instance, use black plastic to conserve soil moisture and control weeds. keep surrounding areas mowed so weeds don't go to seed. Use roundup or a generic brand to keep fence lines clear and spot weed. If you eventually want to have a larger garden, you can prepare the future garden areas by building your soil and diminishing the weed problem by planting cover crops and plowing them under. Install your fence so that you can just add to garden size without having to rip out everything and starting over. Gardening is not easy and requires a lot of work. If you garden smart you can at least have some success, maybe a lot as you solve problems that you will face. If you don't, you will spend every minute you are there trying to maintain a garden and will hate it, fail, and give up. As I said in the beginning,"start small", experience and planning will be your best teacher.