ChatGPT is disappointingly stupid

[Anonymous]

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

Well I've been a lawyer for a decade and it's serving my purposes just fine. It gives me a quick summary and saves me time from writing one myself. It's just picking up language patterns in a pre-existing document and it's good at it. It's also improved a lot in just the past few months. This isn't a static technology. I'm having a lot of fun experimenting, trying out different programs, seeing what they can do and how far they can go before they hit a wall.

At a decade out, you aren’t sophisticated enough to pick up the subtle but significant errors.

There is a reason Harvey is popular with junior associates but not with partners. And it isn’t the lack of sophistication of the partners. It’s the error rates.

NP here: Harvey’s an interesting case. I think the real reason Harvey isn’t that useful for partners is a combination of the fact that its not (yet) capable of handling the partner-level issues that come up in practice, which tend to be a lot of highly context-sensitive judgment calls, and that its writing is only ok by Big Law standards. After playing with it a lot, I don’t use it that much, even though it is amazing what it is capable of if you think about it. I have, however, told our junior associates that they should basically have it open all the time and use it to brainstorm and collaborate on everything, i think its a fabulous educational tool on that level, it does sometimes generate useful insights, and it writes better than all but the best of our new associates. The huge weakness is you can’t use it for research yet, you don’t know what its missing or what its making up without checking, and at least my experience is that the Harvey team intensely disavows this functionality (which is too bad as that would be the most useful thing.). In my experience it’s reasonably good at summarizing documents; other non-Harvey platforms seems to be a little better for what i need. I used o3 to help me get ready for a pitch, which entailed summarizing and organizing publicly available information about a potential client’s operation, and it was astonishingly helpful and fast—but there it didn’t have to be right, exactly, it just had to orient me to context and put me in a position to ask intelligent questions.

[Anonymous]

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

Well I've been a lawyer for a decade and it's serving my purposes just fine. It gives me a quick summary and saves me time from writing one myself. It's just picking up language patterns in a pre-existing document and it's good at it. It's also improved a lot in just the past few months. This isn't a static technology. I'm having a lot of fun experimenting, trying out different programs, seeing what they can do and how far they can go before they hit a wall.

At a decade out, you aren’t sophisticated enough to pick up the subtle but significant errors.

There is a reason Harvey is popular with junior associates but not with partners. And it isn’t the lack of sophistication of the partners. It’s the error rates.

Your unnecessarily insulting language suggests to me that you don’t really understand the potential of this technology and feel sort of threatened by it. I’ve used ChatGPT successfully to summarize long documents, develop talking points, timelines, code, etc. While it definitely occasionally spits out errors, that’s not a problem if you review the work. It’s worth the time saved.

Anyway, not my problem, if I find something that saves me hours a day I’m using it. Sorry you can’t pad your billing as much.

I hire lawyers, and have written my own ChatGPT wrappers. I recognize lawyers who don’t know what they are talking about from a mile away.

You do not understand the errors you are missing, full stop.

I’m not sure why you’re being so aggressive about this? No wonder law firms are so miserable. I explained to you that ChatGPT creates a great starting point- a brief summary, outline, timeline, piece of code. You then build off of that. In my particular job- endlessly useful.

If you’re expecting it to write a whole memo or do a search of case law- yeah, there will be errors there, to the point where it’s not worth it. If you’ve used it a lot you can recognize how it uses language patterns to create summaries and outlines. It can’t do a deep analysis yet, nor can it write a lengthy document of any kind (yet)- things kind of fall apart. But it’s incredibly useful if you recognize the limitations.

[Anonymous]

It failed miserably (score 20%) when I gave it a CPA CPE multiple choice test. lol

[Anonymous]

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

This. It gives you an "answer shaped" block of text, but that is not actually an answer. If you are too uninformed (or lazy) you won't know the difference but you'll be wrong. If you are informed enough to know, it added no value.

It's trash. Worse, it's trash that destroys the environment and steals your data. All you people feeding resumes and travel itineraries in, where do you think that info is going?

I'm at the point where if somebody says "I asked chatGPT ..." I just leave the conversation.

[Anonymous]

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

This. It gives you an "answer shaped" block of text, but that is not actually an answer. If you are too uninformed (or lazy) you won't know the difference but you'll be wrong. If you are informed enough to know, it added no value.

It's trash. Worse, it's trash that destroys the environment and steals your data. All you people feeding resumes and travel itineraries in, where do you think that info is going?

I'm at the point where if somebody says "I asked chatGPT ..." I just leave the conversation.

Couldn’t you say the same about Gmail, Facebook, this website, your google search, and just about everything else on the internet? 🛜

[Anonymous]

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

This. It gives you an "answer shaped" block of text, but that is not actually an answer. If you are too uninformed (or lazy) you won't know the difference but you'll be wrong. If you are informed enough to know, it added no value.

It's trash. Worse, it's trash that destroys the environment and steals your data. All you people feeding resumes and travel itineraries in, where do you think that info is going?

I'm at the point where if somebody says "I asked chatGPT ..." I just leave the conversation.

Couldn’t you say the same about Gmail, Facebook, this website, your google search, and just about everything else on the internet? 🛜

The environmental cost of generative AI is sharply higher than the others you mention.

[Anonymous]

Anonymous wrote:So if you fed it wrong false info, would you be training it to spew incorrect solutions?

garbage in, garbage out. AI trolls the internet, so yea, it could potentially return false information and conspiracy theories if there a lots of posts.

[Anonymous]

It’s kind of like having a college intern. It’s better than me at some stuff, worse at others, and you can’t trust it to know what it does and doesn’t know how to do. But it’s still amazing and I think only a fool would write it off.

[Anonymous]

Anonymous wrote:It’s kind of like having a college intern. It’s better than me at some stuff, worse at others, and you can’t trust it to know what it does and doesn’t know how to do. But it’s still amazing and I think only a fool would write it off.

I don’t write it off. I use it daily. But I also don’t think it’s amazing or magical, and I think it has a shockingly high error rate that the AI marketing people are spending a lot of time trying to hide.

[Anonymous]

Anonymous wrote:It’s kind of like having a college intern. It’s better than me at some stuff, worse at others, and you can’t trust it to know what it does and doesn’t know how to do. But it’s still amazing and I think only a fool would write it off.

It's a useful tool, but you still have to check the output. That NE professor didn't do that . Oops.

[Anonymous]

Grok and Gemini: compare what you get in query with ChatGPT. See which refines better.

[Anonymous]

Anonymous wrote:It failed miserably (score 20%) when I gave it a CPA CPE multiple choice test. lol

Which model failed? Was it with the free version. I thought newer ones pass at almost 90% all these tests.

[Anonymous]

It depends what you are expecting it to do. Take everything with a grain of salt. If I need an exact answer, I'm not going to ask chatgpt. You can't believe everything that comes out of the screen.

[Anonymous]

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:

Anonymous wrote:I’m honestly having a blast using it and I don’t understand why people aren’t figuring out what a great tool AI is if you don’t rely on it to hand you your work.

Here is what I use it for:
- coding to create automated spreadsheets and word templates- I already knew how to do some computer programming honestly but I was not going to make this effort w/o ChatGPT
- Converting data and moving it around, turning documents into tables that I can load into excel and turn into a mini database
- uploading and summarizing long documents like legislation or regulations- it’s accurate at this
- outlining out ideas, organizing my thoughts and pointing out things I missed
- planning out steps for long term projects

It’s a fantastic tool but you’ll notice I’m not asking it to do my job, just augment things I do.

It’s not accurate at the bolded. It only seems accurate to people who don’t have the experience and skill to catch the mistakes it makes. It looks very accurate, but is not actually accurate.

Well I've been a lawyer for a decade and it's serving my purposes just fine. It gives me a quick summary and saves me time from writing one myself. It's just picking up language patterns in a pre-existing document and it's good at it. It's also improved a lot in just the past few months. This isn't a static technology. I'm having a lot of fun experimenting, trying out different programs, seeing what they can do and how far they can go before they hit a wall.

At a decade out, you aren’t sophisticated enough to pick up the subtle but significant errors.

There is a reason Harvey is popular with junior associates but not with partners. And it isn’t the lack of sophistication of the partners. It’s the error rates.

NP here: Harvey’s an interesting case. I think the real reason Harvey isn’t that useful for partners is a combination of the fact that its not (yet) capable of handling the partner-level issues that come up in practice, which tend to be a lot of highly context-sensitive judgment calls, and that its writing is only ok by Big Law standards. After playing with it a lot, I don’t use it that much, even though it is amazing what it is capable of if you think about it. I have, however, told our junior associates that they should basically have it open all the time and use it to brainstorm and collaborate on everything, i think its a fabulous educational tool on that level, it does sometimes generate useful insights, and it writes better than all but the best of our new associates. The huge weakness is you can’t use it for research yet, you don’t know what its missing or what its making up without checking, and at least my experience is that the Harvey team intensely disavows this functionality (which is too bad as that would be the most useful thing.). In my experience it’s reasonably good at summarizing documents; other non-Harvey platforms seems to be a little better for what i need. I used o3 to help me get ready for a pitch, which entailed summarizing and organizing publicly available information about a potential client’s operation, and it was astonishingly helpful and fast—but there it didn’t have to be right, exactly, it just had to orient me to context and put me in a position to ask intelligent questions.

This is consistent with my experience. Harvey is largely being used to train associates but not actually make them more efficient or cost-effective. And it’s very expensive. That doesn’t seem like a successful long-term model for firms: associates aren’t learning as well, jobs aren’t being replaced, and there aren’t costs cut that can be passed to clients.

[Anonymous]

AI has even hurt web searching; half the time the AI summary is useless and search results have decreased in accuracy.