langchain
core: correctly format ImagePromptTemplate when used with a _StringImageMessagePromptTemplate (e.g. HumanMessagePromptTemplate)
#22559
Closed
Overview
Commits
2
Changes
View On GitHub

core: correctly format ImagePromptTemplate when used with a _StringImageMessagePromptTemplate (e.g. HumanMessagePromptTemplate) #22559

thdesc wants to merge 2 commits into langchain-ai:master from thdesc:image_prompt_template_format
thdesc
thdesc334 days ago
  • Description: When formatting an ImagePromptTemplate, only the url, path and detail variables are used (see the format method). However inputs (defined here) cannot have any of these values, since url, path and detail are not allowed as input_variables in ImagePromptTemplate (see here).
    By passing the kwargs directly, it would allow the correct formatting of the ImagePromptTemplate. Here is an example of how it would work:
from langchain_core.prompts import HumanMessagePromptTemplate, PromptTemplate

human_message_prompt_template = HumanMessagePromptTemplate(
    prompt=[PromptTemplate.from_template("Describe the following image"), ImagePromptTemplate()]
)
print(human_message_prompt_template.format(url="https://example.com/image.jpg", detail="auto"))
# Output:
# HumanMessage(content=[{'type': 'text', 'text': 'describe the following image'}, {'type': 'image_url', 'image_url': {'url': 'https://example.com/image.jpg', 'detail': 'auto'}}])
print(human_message_prompt_template.format(path="path/to/image.png", detail="auto"))
# Output:
# HumanMessage(content=[{'type': 'text', 'text': 'describe the following image'}, {'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAAB......QAAIABJRE', 'detail': 'auto'}}])

  • Issues:

  • Dependencies: None

  • Twitter handle: @DeschampsTho5

format image prompt template with kwargs instead of inputs
34184fff
dosubot dosubot added size:XS
vercel
vercel334 days ago (edited 334 days ago)

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
langchain ⬜️ Ignored (Inspect) Visit Preview Jun 5, 2024 7:24pm
dosubot dosubot added 🤖:bug
thdesc Merge branch 'master' into image_prompt_template_format
76988681
thdesc
thdesc334 days ago

At the end, this PR is not useful. I have created another one that is related to the same problem: #22613

thdesc thdesc closed this 334 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
🤖:bug size:XS
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum